BTCSpark: Scalable Analysis of the Bitcoin Blockchain using Spark

There is a large demand in the Bitcoin research ecosystem for high quality, scalable analytic software. Analysis can help developers quantify the risks and benefits of modifications to the Bitcoin protocol, as well as monitor for errant behavior. Historians might use Blockchain analysis to understand how various events impacted on chain activity. Corporations can use analysis to understand their customers better. To quote Madars Virza, Co-Inventor of Zerocash and researcher at MIT CSAIL, “I need to quickly prototype ideas for my research, but parsing the Blockchain for each project is an arduous task, so I’m forced to speculate. The research world is in great need for programmable Blockchain analysis tools.” In order to serve this need, I have developed BTCSpark for my Large Scale Systems (6.S897) final project under Matei Zaharia. BTCSpark is a layer on top of Apache Spark for analyzing the Bitcoin Blockchain. It provides an easy to use, flexible, and good performance environment for researchers and developers to query the Blockchain and to build Blockchain analysis tools. BTCSpark is open source software, in contrast to almost all other user-friendly Blockchain analysis tools available today. BTCSpark can be said to be a forward thinking project. Currently, the Bitcoin Blockchain is 50 GB – certainly feasible to fit it all into memory. However, as the Blockchain grows, and as users of the Blockchain call for it to grow at a faster rate, there will be more and more data. Not only that, but the Blockchain data is in a minimal data format. Fully expanding all of the connections and richness of the data set can require much more memory, even when the underlying set is only 50GB (considering each transaction as a node with potentially